feat: Use WAL Manager #13491
feat: Use WAL Manager #13491
Conversation
This commit changes the RF-1 ingester to use the WAL Manager instead of flushCtx.
| @@ -213,9 +217,18 @@ func (s *stream) storeEntries(ctx context.Context, entries []*logproto.Entry, us | |||
|
|
|||
| bytesAdded += len(entries[i].Line) | |||
| } | |||
| flushCtx.segmentWriter.Append(s.tenant, s.labels.String(), s.labels, entries) | |||
|
|
|||
| res, err := w.Append(wal.AppendRequest{ | |||
There was a problem hiding this comment.
It might be more performant to use a pointer for Append here - unsafe.Sizeof says the AppendRequest is 80B and since it's passed by value it gets copied into the function call.
I'm not sure if this actually leads to any performance benefit in this case but might be worth a benchmark since it's called so often?
There was a problem hiding this comment.
A lot of the time &wal.AppendRequest{} will be turned into wal.AppendRequest{} at compile-time as it's faster to do this copy on the stack instead of allocate it on the heap. It's easier to show with an isolated example.
There was a problem hiding this comment.
Benchmark for using the stack:
go test -gcflags=-N -bench=. bench_test.go
goos: darwin
goarch: arm64
BenchmarkDoRequest-8 573065010 2.098 ns/op
PASS
ok command-line-arguments 2.663s
type AppendRequest struct {
TenantID string
Labels map[string]string
LabelsStr string
}
func doRequest(r AppendRequest) {
_ = r
}
func BenchmarkDoRequest(t *testing.B) {
labels := map[string]string{"foo": "bar"}
labelsStr := "{foo=\"bar\"}"
for i := 0; i < t.N; i++ {
doRequest(AppendRequest{
TenantID: "1",
Labels: labels,
LabelsStr: labelsStr,
})
}
}Benchmark for using the heap:
go test -gcflags=-N -bench=. bench_test.go
goos: darwin
goarch: arm64
BenchmarkDoRequest-8 346227085 3.131 ns/op
PASS
ok command-line-arguments 1.654s
type AppendRequest struct {
TenantID string
Labels map[string]string
LabelsStr string
}
func doRequest(r *AppendRequest) {
_ = r
}
func BenchmarkDoRequest(t *testing.B) {
labels := map[string]string{"foo": "bar"}
labelsStr := "{foo=\"bar\"}"
for i := 0; i < t.N; i++ {
doRequest(&AppendRequest{
TenantID: "1",
Labels: labels,
LabelsStr: labelsStr,
})
}
}| i.flushQueues[0].Enqueue(currentFlushCtx) | ||
| for { | ||
| // Keep adding ops to the queue until there are no more. | ||
| it, _ := i.wal.NextPending() |
There was a problem hiding this comment.
I think we could remove the err return from NextPending if we don't look at it?
There was a problem hiding this comment.
I think so too, I'll do it for both NextPending and Put in another PR. 👍
What this PR does / why we need it:
This commit changes the RF-1 ingester to use the WAL Manager instead of
flushCtx.Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Checklist
CONTRIBUTING.mdguide (required)featPRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.docs/sources/setup/upgrade/_index.mdproduction/helm/loki/Chart.yamland updateproduction/helm/loki/CHANGELOG.mdandproduction/helm/loki/README.md. Example PRdeprecated-config.yamlanddeleted-config.yamlfiles respectively in thetools/deprecated-config-checkerdirectory. Example PR